SAGA-ML: An Active Learning System for Semiautomated Gameplay Analysis

We present SAGA-ML, an active learning system for blackbox software testing. In our approach, labelled examples are obtained by sampling from the space of initial system states and user actions, and then running the blackbox to obtain the labels. This training set is then used to learn a model of the system’s behaviour. Once a model has been learned, it is used to determine where further sampling should occur, increasing the training set for the next iteration of model learning. This intelligent sampling strategy attempts to allocate limited testing resources effectively. Contemporary commercial video games are an example of very complex software systems that have tight development cycles and high reliability standards because aftermarket patching is not always an option. Furthermore, beyond the standard software correctness requirements, they have a more nebulous goal of “enjoyability” that is essentially impossible to specify or measure exactly. It is up to the designer to decide whether the game’s behaviour is appropriate and the data collected can only assist them in the evaluation. In this context, machine learning serves a dual role where the learned model is used to summarize the game’s behaviour for the developer, as well as directing the sampling of additional points. We demonstrate our analysis tool, SAGA-ML (semiautomated gameplay analysis by machine learning), which is game independent and treats the game engine as a black box, and the SoccerViz visualization tool, a game-specific component that displays the learned models in a more comprehensible format. SAGA-ML has correctly discovered socalled “sweet spots” in the game (situations where it is too easy to score) and we demonstrate how these can be easily detected and explored by the developer using the SoccerViz tool. Here we will give a brief overview of the gameplay analysis task, the SAGA-ML architecture, and the SoccerViz tool. More detailed descriptions are available of our work on the gameplay analysis problem (Southey et al. 2005) and active learning as a software testing methodology (Xiao et al. 2005).